Apparatus and method for generating a fully focused image by using a camera equipped with a multi-color filter aperture

ABSTRACT

Provided are an apparatus and method for generating a fully focused image. A depth map generation unit generates a depth map of an input image obtained by a multiple color filter aperture (MCA) camera. A channel shifting &amp; alignment unit extractes subimages which include objects with same focal distance based on the depth map, and performing color channel alignment and removing out-of-focus blurs for each subimages obtained from the depth map. An image fusing unit fuses the subimages to generate a fully focused image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.61/410,4230, filed Nov. 5, 2011, the contents of which are herebyincorporated by reference

BACKGROUND

The present invention relates to an apparatus and method for generatinga fully focused image and, more particularly, to an apparatus and methodfor generating a fully focused image by using a camera equipped with amulti-color filter aperture.

Demand for digital auto-focusing techniques is rapidly increasing inmany visual applications, such as camcorders, digital cameras, and videosurveillance systems. Conventional cameras have come a long way indealing with problems associated with focal settings and blur. Eventhough several steps have been taken, focal blur caused by varyingdistance of the object from the lens has been something that theconventional cameras still have to deal with. With focus set at near,mid or far regions of the scene, the captured image tends to have onlythat particular region in focus where as the remaining regions tend tobe in out-of-focus. To solve this problem, post-processing steps in theform of blur restoration and multiple image fusion have been proposed todeal with the focusing problem.

Recently computational cameras have been developed that are capable ofcapturing additional information from the scene which when combined withpost-processing can overcome several drawbacks of the imagingapplications including: refocusing, increased dynamic range,depth-guided editing, variable lighting and reflectance, and so on.

The idea of using a multiple aperture lens has been previously proposedusing micro lens array and wave front coding. However, the quality ofimages obtained by these optical designs is fundamentally inferior to acamera system with a large single lens. And, the resolution of thesesmall lens arrays is severely limited by diffraction. More recentmethods include single-lens multi-view image capture. This multiplefilter aperture (FA) model uses parallax cues instead of defocus cuesand requires only color filters as additional optical elements to thelens without requiring multiple exposures.

Meanwhile, extensive work has been done using fusion andrestoration-based methods for removal of out-of-focus blur in images.Fusion algorithms using DCT, pyramids, and wavelets have been proposedto name a few where as restoration algorithms include blindde-convolution with no priori information as well as with PSFestimation.

Also, depth map algorithms have been extensively applied to stereovision where the disparity estimate is computed. as a correspondencemeasure through camera displacement. Shape from focus can also estimatedepth from a sequence of images taken by a single camera at differentfocus levels. Shape from focus methods employ spatial criteria includinggray level variance (GLV), sum modified Laplacian (SML), Tanenbaum, meanmethod, curvature focal measure, and so forth.

SUMMARY

The present invention is directed to providing an apparatus and methodfor generating a fully focused image which captured by a multiple colorfilter aperture camera.

The present invention is also directed to providing a non-transitorycomputer readable medium recording a program for executing in a computera method for generating a fully focused image.

According to an aspect of the present invention, there is provided anapparatus for generating a fully focused image includes: a depth mapgeneration unit generating a depth map of an input image obtained by amultiple color filter aperture (MCA) camera; a channel shifting &alignment unit extracting subimages which include objects with samefocal distance based on the depth map, and performing color channelalignment and removing out-of-focus blurs for each subimages obtainedfrom the depth map; and an image fusing unit fusing the subimages togenerate a fully focused image.

According to another aspect of the present invention, there is provideda method for generating a fully focused image includes: (a) generating adepth map of an input image obtained by a multiple color filter aperture(MCA) camera; (b) extracting subimages which include objects with samefocal distance based on the depth map; (c) performing color channelalignment and removing out-of-focus blurs for each subimages obtainedfrom the depth map; and (d) fusing the subimages to generate a fullyfocused image.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings(s) will be provided by the Office upon request andpayment of the necessary fee. The above and other objects, features andadvantages of the present invention will become more apparent to thoseof ordinary skill in the art by describing in detail exemplaryembodiments thereof with reference to the accompanying drawings, inwhich:

FIG. 1 is a block diagram showing the configuration of an apparatus forgenerating a fully focused image according to an exemplary embodiment;

FIGS. 2A to 2C are input images and corresponding depth maps extractedusing color channel dependency.

FIGS. 3A to 3D are views for explaining images formed by using anaperture with three openings;

FIG. 4 is a schematic and functional illustration of the multiple filteraperture model;

FIG. 5 is a flow chart illustrating the process of a method forgenerating a fully focused image according to an exemplary embodiment;

FIGS. 6A to 6C are input images with different focal lengths; and

FIGS. 7A to 7C are fully focused images obtained by using the methodaccording to an exemplary embodiment from input images shown in FIGS. 3Ato 3C.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, an apparatus and method for enhancing image quality of animage according to exemplary embodiments will be described withreference to the accompanying drawings. Throughout the specification andclaims, unless explicitly described to the contrary, the word “comprise”and variations such as “comprises” or “comprising”, will be understoodto imply the inclusion of stated elements but not the exclusion of anyother elements. Terms such as ‘unit’, ‘module’, ‘block’, or the like,described in the specification may refer to a unit for processing atleast one function or operation, which can be implemented by hardware,software, or a combination of hardware and software.

FIG. 1 is a block diagram showing the configuration of an apparatus forgenerating a fully focused image according to an exemplary embodiment ofthe present invention.

Referring to FIG. 1, an apparatus for generating a fully focused image100 includes a depth map generation unit 110, a channel shifting &alignment unit 120, an image fusing unit 130, and a smoothing unit 140.

The depth map generation unit 110 generates a depth map of an inputimage obtained by the multiple color filter aperture (MCA) camera. FIGS.2A to 2D are views for explaining images formed by using an aperturewith three openings. As shown in FIG. 2A, the MCA camera uses threeopenings with R, G, and B filters. With reference to FIGS. 2B to 2D, amain advantage of the MCA camera is that it provides additional depthinformation which can be estimated from the direction and amount ofcolor deviation from the optical axis. When an object is located on theout-focused position, the MCA camera results in out-of-focus blur aswell as color shifting.

A fully focused image can be generated from n subimages using estimateddepth map. To generate the depth map, the depth map generation unit 110decomposes the input image into regions of homogeneous color. Here, theperimeter of the each region can be obtained from the edge informationwhich in turn supports the disparity estimation. For this, we assume thepixel colors within a local window w(x,y) belong to one cluster and usethe magnitude of the each cluster elongation as the correspondencemeasure. More specifically, we consider a set P₁(x,y;d) of pixel colorswith hypothesized disparity d as Equation 1 shown below:

P ₁(x,y;d)={(I _((R))(s+d,t), I _((G))(s,t−d), I_((B))(s−d,t))|(s,t)∈w(x,y)}  Equation 1

Here, I_((R)), I_((G)), and I_((B)) represent red, green, and blue colorchannel images, respectively and (s+d,t), (s,t−d), and (s−d,t) are shiftvectors for each color channel images.

And, the depth map generation unit 110 searches for d that minimizes thefollowing Equation 2 shown below for color alignment measure:

L(x,y;d)=λ₀λ₁λ₂/σ_(r) ²σ_(g) ²σ_(b) ²  Equation 2

Here, λ₀, λ₁, and λ₂ represent the eigenvalues of the covariance matrixΣ of the color distribution P₁(x,y;d), respectively, and σ_(r) ², σ_(g)², and σ_(b) ² are the diagonal elements of the covariance matrix Σ.

From Equations 1 and 2, the depth map generation unit 110 can obtain anabstract disparity map in the predetermined disparity search range (forexample, [−10, 10]) which can be used to develop the error energy matrixand calculate the depth map. However, the depth map extracted using theabove-mentioned method alone is insufficient for the proper extractionof focus maps needed to generate the fully focused image. Hence,segmentation as an additional criterion for more accurate extraction ofthe depth map α(x,y) as the following Equation 3:

α(x,y)=L(x,y)∩M(x,y)  Equation 3

Here, M(x,y) represents the mean-shift segmentation result.

FIGS. 3A to 3C are input out-of-focus images captured using the MCAcamera and corresponding depth maps extracted using color channeldependency.

Referring to FIGS. 3A to 3C, each input image shown left in each figureis focused at one object, that is deer, spring, and man, respectively.Therefore, other objects in each image suffer from out-of-focus blurs.The depth maps shown right in each figure are composed of severalobjects of which colors are different. In the depth map, objects havingsame color are assumed to be same distance from the lens of the MCAcamera.

The channel shifting & alignment unit 120 performs color channelalignment and remove the out-of-focus blur for each subimages obtainedfrom the depth map. When the center of the aperture is not aligned onthe optical axis, convergence is made off the optical axis, whosespecific location depends on the distance between the lens and anobject. The R, G, and B filters on FA are arranged so that theirdisplacement with respect to center of the lens aligns with the rowcolumn displacement of the image sensor as shown in FIG. 4. Referring toFIG. 4, for objects located at near-, far-, and in-focused distance, theconvergence pattern is split into three channels, thereby, retaining theinformation of circle of confusion (COC) or point spread function (PSF)within the corresponding color plane.

By this arrangement, a scene point nearer or farther than the focuseddepth is captured as shift in R, G and B channels. The main advantage ofthe FA model is that it can provide an alternative method for the blurestimation in auto-focusing applications. Images acquired by using aconventional optical system have defocusing blur caused by a specificPSF. On the other hand the proposed multiple FA model the auto-focusingproblem turns in to the alignment of R, G, and B channels with variousdepths of field. For shifting and aligning color channels we need tofind the optimal pixel-of-interest (POI) at different positions in theimage according to their focal measures. The POI can be referred to as afocal point pixel, around which channel shifting and alignment iscarried out. For a given region, we then select the focal point pixeleither from the center of the region or the pixel with the lowest focusmeasure. Similar operations repeat for differently selected focal pointregions.

For a given particular image acquired by using the MCA camera configuredas shown in FIG. 4, the out-of-focus blur was just confined to channelson either side of the green, thereby, suffering minimal distortion dueto the central sensor position whereas the red and the blue channelshave maximal blur distortion. Therefore, the green channel can be usedas the reference and the red and the blue channels can be aligned to thegreen by using Equation 4:

I _((R,G,B)) =[S _((r,c))(I _((R)), I _((B))), I _((G))]  Equation 4

Where, I_((R,G,B)) represents the input image and S_((r,c)) representsthe shift operator of the corresponding shift vector (r,c) thatrepresents the amount of shift in row and column directions for therespective red and blue channels with respect to the reference focalpoint on the green channel.

If the shift vectors are not identical, Equation 4 can be generalized asfollow:

I _((R,G,B)) =[S _((r1,c1))(I _((R))), ∈_((r2,c2))(I _((B))), I_((G))]  Equation 5

Where, I_((R,G,B)) represents the input image and S_((r1,c1)) andS_((r2,c2)) represent the shift operators of the corresponding shiftvectors (r₁,c₁) for red channel and (r₂,c₂) for blue channel.

According to Equation 5, for aligning blue channel with green channel,the pixels have to be shifted in upward direction and toward left ordiagonally to left and vice versa for red channel.

In the meantime, the input image can be represented as follow:

I_((R,G,B))={I_((R) ₁ _(,G) ₁ _(,B) ₁ ₎ ^(fp1), . . . , I_((R) _(n)_(,G) _(n) _(,B) _(n) ₎ ^(fpn)}  Equation 6

Where, I_((R,G,B)) represents the input image and I_((R) ₁ _(,G) ₁ _(,B)₁ ₎ ^(fp1), . . . , I_((R) _(n) _(,G) _(n) _(,B) _(n) ₎ ^(fpn) representsubimages at varying focal points.

The channel shifting & alignment unit 120 extracts subimages based onthe depth map. Each subimage includes objects which have same focaldistance, that is to say, which have same color in the depth map. Andthe channel shifting & alignment unit 120 aligns two color channels (redand blue) to a reference color channel (green) for each subimage. Forthis, the channel shifting & alignment unit 120 obtains fully focusedsubimages by using Equation 7 based on the depth map.

(I _((R) ₁ _(,G) ₁ _(,B) ₁ ₎ ^(FR1) , . . . , I _((R) _(n) _(,G) _(n)_(,B) _(n) ₎ ^(FRn))=(L,M)⊂(I _((R) ₁ _(,G) ₁ _(,B) ₁ ₎ ^(fp1) , . . . ,I _((R) _(n) _(,G) _(n) _(,B) _(n) ₎ ^(fpn))  Equation 7

Where, I_((R) ₁ _(,G) ₁ _(,B) ₁ ₎ ^(FR1), . . . , I_((R) _(n) _(,G) _(n)_(,B) _(n) ₎ ^(FRn) represent fully focused subimages.

The image fusing unit 130 fuses subimages to generate a fully focusedimage. For this, the image fusing unit 130 combines different regionsfrom different channel shifted regions using the depth map informationas follow:

I _((R,G,B))=(I _((R) ₁ _(,G) ₁ _(,B) ₁ ₎ ^(FR1) + . . . +I _((R) _(n)_(,G) _(n) _(,B) _(n) ₎ ^(FRn))  Equation 8

Where, I_((R,G,B)) ^(F) represents the fully focused image.

The smoothing unit 140 divides the image into a high frequency regionand a low frequency region by using a spatially adaptive noise smoothingalgorithm based on an alpha map in order to enhance image quality of thefully focused image.

FIG. 5 is a flow chart illustrating the process of a method forgenerating a fully focused image according to an exemplary embodiment.

Referring to FIG. 5, the depth map generation unit 110 establishes a setof pixel colors with hypothesized disparity d and searches for d thatminimizes color alignment measure by using Equations 1 and 2 in stepS510. Next, the depth map generation unit 110 generates depth map bycombining the abstract disparity map and the mean-shift segmentationresult by using Equation 3 in step S520. Next, the channel shifting &alignment unit 120 obtains subimages from the depth map and generatefully focused subimages by using the depth map by using Equation 7 instep S530. Next, the fusing unit 130 generates a fully focused image byfusing fully focused subimages in step S540. Finally, the smoothing unit140 enhances the quality of the fully focused image by smoothing in stepS550. The step S550 can be omitted.

For the experiments, a commercial gelatin filter (Kodak-WrattenFilter—G-58, B-47, and R-25) with sensors representing red, green, andblue spectral wavelengths is used. FIG. 6( a) represents a typical FAmultiple object image captured with focus on ‘spring’ object on left.FIGS. 6( b) and 6(c) represent channel shifting performed to shift thefocus from ‘spring’ to ‘cowboy’ followed by ‘robot’ using FIG. 6( a). Inorder to combine FIGS. 6( a)-6(c) to a single image with full focus thedepth map estimation has been used as shown in FIGS. 3A to 3C. By usingthe generated depth map we can extract boundary of each object which inturn retrieve and combine pixels from various channel shifted imagesshown in FIGS. 7A to 7C, of which input images and corresponding depthmaps are shown in FIGS. 3A to 3C, respectively.

The result of the present invention was compared against standardrestoration and fusion-based methods as shown in Table 1.

TABLE 1 AF method Priori Mode Input Operation RMSE PSNR Wiener FilterPSF Gray 1 Pixel 12.35 23.36 Iterative Filter NIL Gray 1 Pixel 8.5626.32 Pyramid Fusion NIL Gray, At least 2 Window 5.68 28.42 Color andPixel Wavelet Fusion NIL Gray, At least 2 Window 5.02 29.95 Color andPixel Present NIL Color 1 Window 8.06 26.41 Invention and Pixel

Another comparison in the sense of pixel error count (PEC) and disparityerror map (DEM) are shown in Tables 2 and 3. PEC is obtained bycalculating the number of mis-classified pixels with ground truthsegmentation map. Shape from focus measures including sum modifiedLaplacian (SML), gray level variance (GLV), Tanenbaum and Tanengrad wereused in PEC comparison.

TABLE 2 Present Test SML Tanenbaum GLV Tenengrad Invention Deer 1.050.95 1.35 0.99 0.52 Toys 0.99 0.61 1.30 0.93 0.41 Sim 1.12 0.82 1.170.91 0.50 Doll 0.82 0.67 1.03 0.68 0.49

TABLE 3 Present Test SAD GRAD Color Bayes Invention Deer 14.56 12.107.20 6.30 6.50 Toys 18.31 14.32 8.60 8.40 6.21 Sim 12.08 11.47 7.01 5.645.40 Doll 12.32 12.19 9.18 7.82 6.11

It can be seen that the proposed depth map had comparable results withTanenbaum for PEC but outperforming other measures significantly. TheDEM was used to find the disparity error average for pre-defined range[−10 10] when compared to ground truth data. Stereo vision methodsincluding sum and gradient absolute differences (SAD and GRAD), colorand Bayes disparity were tested. In case of DEM the performance showedvast improvements over SAD and GRAD measures whereas comparativelycompetitive with color and Bayes disparity.

The present invention is ideal for situations when the focal range of ascene is distributed over varying distance from the camera.

The present invention can be embodied as computer readable codes on acomputer readable recording medium. The computer readable recordingmedium is any data storage device that can store data which can bethereafter read by a computer system. Examples of the computer readablerecording medium include read-only memory (ROM), random-access memory(RAM), CD-ROMs, magnetic tapes, floppy disks, optical data storagedevices, and carrier waves (such as data transmission through theInternet). The computer readable recording medium can also bedistributed over network coupled computer systems so that the computerreadable code is stored and executed in a distributed fashion.

Although the apparatus and method for generating a fully focused imagecaptured by using MCA camera have been described with reference to thespecific embodiments, they are not limited thereto. Therefore, it willbe readily understood by those skilled in the art that variousmodifications and changes can be made thereto without departing from thespirit and scope of the present invention defined by the appendedclaims.

1. An apparatus for generating a fully focused image, the apparatuscomprising: a depth map generation unit generating a depth map of aninput image obtained by a multiple color filter aperture (MCA) camera; achannel shifting & alignment unit extracting subimages which includeobjects with same focal distance based on the depth map, and performingcolor channel alignment and removing out-of-focus blurs for eachsubimages obtained from the depth map; and an image fusing unit fusingthe subimages to generate a fully focused image.
 2. The apparatus ofclaim 1, wherein the depth map generation unit generates the depth mapby establishing a set of pixel colors with hypothesized disparity d asEquation 1 shown below, and searches for d that minimizes an abstractdisparity map represented by following Equation 2:P ₁(x,y;d)={(I _((R))(s+d,t), I _((G))(s,t−d), I_((B))(s−d,t))|(s,t)∈w(x,y)}  Equation 1L(x,y;d)=λ₀λ₁λ₂/σ_(r) ²σ_(g) ²σ_(b) ²  Equation 2, here, I_((R)),I_((G)), and I_((B)) represent red, green, and blue color channelimages, respectively, (s+d,t), (s,t−d), and (s−d,t) are shift vectorsfor each color channel images, λ₀λ₁, and λ₂ represent the eigenvalues ofthe covariance matrix Σ of the color distribution P₁(x,y;d),respectively, w(x,y) is a local window, and σ_(r) ², σ_(g) ², and σ_(b)² are the diagonal elements of the covariance matrix Σ.
 3. The apparatusof claim 2, wherein the depth map generation unit generates the depthmap by combining the abstract disparity map and a mean-shiftsegmentation result.
 4. The apparatus of claim 1, further comprising: asmoothing unit smoothing the fully focused image to enhance imagequality.
 5. A method for generating a fully focused image, the methodcomprising: (a) generating a depth map of an input image obtained by amultiple color filter aperture (MCA) camera; (b) extracting subimageswhich include objects with same focal distance based on the depth map;(c) performing color channel alignment and removing out-of-focus blursfor each subimages obtained from the depth map; and (d) fusing thesubimages to generate a fully focused image.
 6. The method of claim 5,wherein step (a) comprises: (a1) establishing a set of pixel colors withhypothesized disparity d as Equation 1 shown below; and (a2) searchingfor d that minimizes an abstract disparity map represented by followingEquation 2:P ₁(x,y;d)={(I _((R))(s+d,t), I _((G))(s,t−d), I_((B))(s−d,t))|(s,t)∈w(x,y)}  Equation 1L(x,y;d)=λ₀λ₁λ₂/σ_(r) ²σ_(g) ²σ_(b) ²  Equation 2 here, I_((R)),I_((G)), and I_((B)) represent red, green, and blue color channelimages, respectively, (s+d,t), (s,t−d), and (s−d,t) are shift vectorsfor each color channel images, λ₀, λ₁, and λ₂ represent the eigenvaluesof the covariance matrix Σ of the color distribution P₁(x,y;d),respectively, w(x,y) is a local window, and σ_(r) ², σ_(g) ², and σ_(b)² are the diagonal elements of the covariance matrix Σ.
 7. The method ofclaim 6, wherein step (a) further comprises (a3) generating the depthmap by combining the abstract disparity map and a mean-shiftsegmentation result.
 8. The method of claim 5, further comprising: (e)smoothing the fully focused image to enhance image quality.
 9. Anon-transitory computer readable medium storing a program for executingthe method for managing digital rights according to claim 5 in acomputer.